How to execute only the specific failed tests. By using 'IRetryAnalyzer' we can re-run the failed tests for x number of time. As mentioned here Restart failed test case automatically. I have also implemented iTestListener to make the tests count more meaning full by following Retry Only failed Tests and update test run count by implementing 'ITestListener'
Is there any way to Re-run ONLY specific failed tests.
Example: We need to execute only tests which are failed because of NoSuchElementException and TimeoutException.
Please find the below screen shot where total 8 tests are failed and there are 6 tests which are failed because of NoSuchElementException-1 and TimeoutException-5.
Please help.
You can try out by checking the result of your tests like:
#Override
public boolean retry(ITestResult result) {
try {
if (result.getThrowable().toString()
.contains("NoSuchElementException")) // Checking for specific reasons of failure
if (retryCount < maxRetryCount) {
retryCount++;
return true;
}
return false;
} catch (Exception e) {
return false;
}
}
Since every result has an attribute m_throwable in case of exception has occurred, you can use it to get your task done in Retry class.
Related
Scenario: Below negative scenario to be tested during Integration test. currently the test case getting failed due exit and not reaching to the test method.
example :
private void method1(int a){
try{
if(a == 0){
throw exception();
}else{
---
}
}catch(exceptionclass e){
System.exit(1);
}
}
Sound like a bad smell to me that calling a method on a object can cause JVM to exist. Normally it should be done in the main method.
So I would refactor your codes such that your testing object will throw a kind of Exception to indicate that some kind of fatal error happens such that the main method can catch it and terminate the JVM.
Then you can simply test that if it will throw this Exception from your test case.
In my framework - , Test & PageObjModel classes are there. incase any method is not executed because of wrong xpath or other any issue the Test is terminating but in the Report-Console it is showing all the Testcase are PASS.
Any issue in the Action methods of PageObjModel, some times it will skip that particular line of code and continue execution. sometimes entire test is terminating. in both the case Report-Console it is showing all the Testcase are PASS i.e No failures of Testcases.
Please have a look into below example of code and suggest me to fail the TC incase of failures.
I tried to include Assert.fail(), Assert.true(fail) in the "catch" block, though it is showing PASS results.
PageObjModel:
class A{
// list of xpaths
public static fibal String Login_xpath = "//*[#id='login'];
public void login(){
try{
// All Action methods are define here
}catch (Exception e){
e.printStackTrace();
//Assert.assertTrue(false);
Assert.fail();
} } }
Test class:
#Test
public void loginTest(){
//calling methods
xx.login();
}
This is just sample code. please help me where can I put Assertions to fail the TC when failure occurs in the code
If we are using try-catch in your code, please add throw as well. otherwise, it won't inform testng about the failure.
catch (AssertionError e) {
e.printStackTrace();
takeSnapShot(action);
throw e;
}
I am running cucumber tests in the TestNG framework using the maven command. Daily I am executing the test cases from Jenkins and generating the cucumber report in Jenkins. (using cucumber report plugin)
I am looking for a solution to re-run the failed test cases in Jenkins and it should give the final report.
Please provide me the approach to achieve this.
On of the simple way is, use IRetryAnalyzer in TestNG. It will re-run the failed test case.
In final report if re-run passed then i will display as passed (initially failed one display as skipped)
if re-run also failed, then marked as failure.
example:
public class Retry implements IRetryAnalyzer {
private int count = 0;
private static int maxTry = 3;
#Override
public boolean retry(ITestResult iTestResult) {
if (!iTestResult.isSuccess()) { //Check if test not succeed
if (count < maxTry) { //Check if maxtry count is reached
count++; //Increase the maxTry count by 1
iTestResult.setStatus(ITestResult.FAILURE); //Mark test as failed
return true; //Tells TestNG to re-run the test
} else {
iTestResult.setStatus(ITestResult.FAILURE); //If maxCount reached,test marked as failed
}
} else {
iTestResult.setStatus(ITestResult.SUCCESS); //If test passes, TestNG marks it as passed
}
return false;
}
}
Add in Testng.xml file
You can add test to also
#Test(retryAnalyzer = Retry.class)
I started working with TESTNG for the first time few days back.
I implemented a retry analyzer using the IRetryAnalyzer interface which retries a failed test method for a maximum of 3 times.
I have a test method called retryInvoker() which fails twice and succeeds the third time.
The way TestNG reports this is 3 separate test runs, 2 of which failed and 1 succeeded, which is correct actually.
Although I just wish to capture it as a single test run, which ultimately succeeded (if it did before the maximum allowed retries ended, which in this case were 3). Or even if it didn't succeed after 3 retries, I wish to report it as a single test run that failed instead of 4 separate test runs, all of which failed.
Any leads on this?
You could try the approach, suggested in this SO answer, removing the duplicates from the test context in case the same method is found either in failed or passed tests:
#Overrride
public void onFinish(ITestContext context) {
Iterator<ITestResult> failedTestCases = context.getFailedTests().getAllResults().iterator();
while (failedTestCases.hasNext()) {
System.out.println("failedTestCases");
ITestResult failedTestCase = failedTestCases.next();
ITestNGMethod method = failedTestCase.getMethod();
if (context.getFailedTests().getResults(method).size() > 1) {
System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
} else {
if (context.getPassedTests().getResults(method).size() > 0) {
System.out.println(
"failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
}
}
}
}
My code for handling my pipeline result looks like this (some snipped for brevity's sake):
PipelineResult result = pipeline.run();
switch (result.getState()) {
case DONE: {
handleDone();
break;
}
case FAILED: {
handleFailed();
break;
}
case CANCELLED: {
handleCancelled();
break;
}
case UNKNOWN:
case RUNNING:
case STOPPED: {
handleUnknownRunningStopped();
break;
}
default: {
assert false;
throw new IllegalStateException();
}
}
However, I've noticed that instead of returning a value of the enum PipelineResult.State for e.g. FAILED or CANCELLED, an exception is thrown:
For a failed job, a DataflowJobExecutionException is thrown
For a cancelled job, a DataflowJobCancelledException is thrown
What is the correct way (programmatically) to handle the result of a pipeline?
Both DataflowPipelineRunners return the PipelineResult that allows you to query the current status of the pipeline. The DataflowPipelineRunner returns a PipelineResult immediately upon job submission, but the BlockingDataflowPipelineRunner doesn't return it until the job has completed.
In addition, the BlockingDataflowPipeline runner throws an exception if the job does not complete successfully -- since you've specified a blocking run(), we assume you want to know if something goes wrong. So if you've hard coded the Blocking Runner, then relying on an exception is an easy way to handle failure.
Note that the code snippet you wrote uses the more general PipelineResult option, but won't work with the non-blocking runner, since that will return a result while the job is still running.