I am running cucumber tests in the TestNG framework using the maven command. Daily I am executing the test cases from Jenkins and generating the cucumber report in Jenkins. (using cucumber report plugin)
I am looking for a solution to re-run the failed test cases in Jenkins and it should give the final report.
Please provide me the approach to achieve this.
On of the simple way is, use IRetryAnalyzer in TestNG. It will re-run the failed test case.
In final report if re-run passed then i will display as passed (initially failed one display as skipped)
if re-run also failed, then marked as failure.
example:
public class Retry implements IRetryAnalyzer {
private int count = 0;
private static int maxTry = 3;
#Override
public boolean retry(ITestResult iTestResult) {
if (!iTestResult.isSuccess()) { //Check if test not succeed
if (count < maxTry) { //Check if maxtry count is reached
count++; //Increase the maxTry count by 1
iTestResult.setStatus(ITestResult.FAILURE); //Mark test as failed
return true; //Tells TestNG to re-run the test
} else {
iTestResult.setStatus(ITestResult.FAILURE); //If maxCount reached,test marked as failed
}
} else {
iTestResult.setStatus(ITestResult.SUCCESS); //If test passes, TestNG marks it as passed
}
return false;
}
}
Add in Testng.xml file
You can add test to also
#Test(retryAnalyzer = Retry.class)
Related
I started working with TESTNG for the first time few days back.
I implemented a retry analyzer using the IRetryAnalyzer interface which retries a failed test method for a maximum of 3 times.
I have a test method called retryInvoker() which fails twice and succeeds the third time.
The way TestNG reports this is 3 separate test runs, 2 of which failed and 1 succeeded, which is correct actually.
Although I just wish to capture it as a single test run, which ultimately succeeded (if it did before the maximum allowed retries ended, which in this case were 3). Or even if it didn't succeed after 3 retries, I wish to report it as a single test run that failed instead of 4 separate test runs, all of which failed.
Any leads on this?
You could try the approach, suggested in this SO answer, removing the duplicates from the test context in case the same method is found either in failed or passed tests:
#Overrride
public void onFinish(ITestContext context) {
Iterator<ITestResult> failedTestCases = context.getFailedTests().getAllResults().iterator();
while (failedTestCases.hasNext()) {
System.out.println("failedTestCases");
ITestResult failedTestCase = failedTestCases.next();
ITestNGMethod method = failedTestCase.getMethod();
if (context.getFailedTests().getResults(method).size() > 1) {
System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
} else {
if (context.getPassedTests().getResults(method).size() > 0) {
System.out.println(
"failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
}
}
}
}
This question already has an answer here:
Karate framework retry until not working as expected
(1 answer)
Closed 2 years ago.
Retry mechanism in karate testing framework How to retry tests on failure in karate testing framework like Junit and TestNG.
something like
public class Retry implements IRetryAnalyzer {
private int count = 0;
private static int maxTry = 3;
#Override
public boolean retry(ITestResult iTestResult) {
if (!iTestResult.isSuccess()) { //Check if test not succeed
if (count < maxTry) { //Check if maxtry count is reached
count++; //Increase the maxTry count by 1
iTestResult.setStatus(ITestResult.FAILURE); //Mark test as failed
return true; //Tells TestNG to re-run the test
} else {
iTestResult.setStatus(ITestResult.FAILURE); //If maxCount reached,test marked as failed
}
} else {
iTestResult.setStatus(ITestResult.SUCCESS); //If test passes, TestNG marks it as passed
}
return false;
}
}
It works for me on version 0.9.5.RC5 . But, maybe this is one of the before-mentioned "workarounds"?
All you do is something like this, which defaults to 3 attempts:
* retry until responseStatus == 404
When method get
As of now this is an open feature request: https://github.com/intuit/karate/issues/247
But there are multiple work arounds. You may get some ideas if you look at the polling example: https://github.com/intuit/karate/blob/master/karate-demo/src/test/java/demo/polling/polling.feature
I was following several different Web Sites explaining how to use RetryAnalyzer (they all say basically the same thing, but I checked several to see whether there was any difference). I implemented as they did in the samples, and deliberately caused a failure the first run (which ended up being the only run). Even though it failed, the test was not repeated. I even put a breakpoint inside the analyzer at the first line (res = false). which never got hit. I tell it to try 3 times but it did not retry at all. Am I missing something?
My sample is below: Is it something to do with setting counter = 0? But the "res = false" at least should get hit?
public class RetryAnalyzer implements IRetryAnalyzer {
int counter = 0;
#Override
public boolean retry(ITestResult result) {
boolean res = false;
if (!result.isSuccess() && counter < 3) {
counter++;
res = true;
}
return res;
}
}
and
#Test(dataProvider = "dp", retryAnalyzer = RetryAnalyzer.class)
public void testA(TestContext tContext) throws IOException {
genericTest("A", "83701");
}
The test usually passes. I deliberately caused it to fail but it did not do a retry. Am I missing something?
===============================================
Default suite
Total tests run: 1, Failures: 1, Skips: 0
Try adding alwaysRun = true to your test method decorator.
#Test(dataProvider = "dp", retryAnalyzer = RetryAnalyzer.class, alwaysRun = true)
public void testA(TestContext tContext) throws IOException {
genericTest("A", "83701");
}
Also, before retrying, you may want to restart your driver instance, so that you start clean with your test. Otherwise, your second run will execute in the same browser instance.
Just do a driver.Quit() following by a reinstantiation of the browser driver.
RetryAnalyzer class has to be public. Also, if it is an inner class, it should be static. TestNg silently ignores retryAnalyzer otherwise.
How to execute only the specific failed tests. By using 'IRetryAnalyzer' we can re-run the failed tests for x number of time. As mentioned here Restart failed test case automatically. I have also implemented iTestListener to make the tests count more meaning full by following Retry Only failed Tests and update test run count by implementing 'ITestListener'
Is there any way to Re-run ONLY specific failed tests.
Example: We need to execute only tests which are failed because of NoSuchElementException and TimeoutException.
Please find the below screen shot where total 8 tests are failed and there are 6 tests which are failed because of NoSuchElementException-1 and TimeoutException-5.
Please help.
You can try out by checking the result of your tests like:
#Override
public boolean retry(ITestResult result) {
try {
if (result.getThrowable().toString()
.contains("NoSuchElementException")) // Checking for specific reasons of failure
if (retryCount < maxRetryCount) {
retryCount++;
return true;
}
return false;
} catch (Exception e) {
return false;
}
}
Since every result has an attribute m_throwable in case of exception has occurred, you can use it to get your task done in Retry class.
I am using Selenium webdriver, in Java with TestNG to run an X amount of test cases.
What I would like, is for any test case to automatically restart (either from starting or from point of failure), as soon as it fails.
I know TestNG framework has the following method
#Override
public void onTestFailure(ITestResult tr) {
log("F");
}
but I do not know how to find out which testcase it was and then how would I restart it.
I wanted to see an example with actual code in it and found it here:
Restarting Test immediately with TestNg
Observe how the below tests will each be re-run once as soon as the failure happens.
import org.testng.Assert;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.annotations.Test;
public class Retry implements IRetryAnalyzer {
private int retryCount = 0;
private int maxRetryCount = 1;
public boolean retry(ITestResult result) {
if (retryCount < maxRetryCount) {
retryCount++;
return true;
}
return false;
}
#Test(retryAnalyzer = Retry.class)
public void testGenX() {
Assert.assertEquals("james", "JamesFail"); // ListenerTest fails
}
#Test(retryAnalyzer = Retry.class)
public void testGenY() {
Assert.assertEquals("hello", "World"); // ListenerTest fails
}
}
From testng.org
Every time tests fail in a suite, TestNG creates a file called testng-failed.xml in the output directory. This XML file contains the necessary information to rerun only these methods that failed, allowing you to quickly reproduce the failures without having to run the entirety of your tests.
If you want to rerun the test exactly after the failure you need to call the method that failed. You can get that method name from ITestResult object.
If you want to rerun all the failed test cases together, then you can give the testng-failed.xml as input xml after the first execution.