Retry Logic - retry whole class if one tests fails - selenium - selenium

Following are the classes used to implement retry logic
TestRetry Class:
public class TestRetry implements IRetryAnalyzer {
int counter=0;
int retryLimit=2;
#Override
public boolean retry(ITestResult result) {
if (counter < retryLimit) {
TestReporter.logStep("Retrying Test " +result.getName()+" for number of times: "+(counter+1));
counter++;
return true;
}
return false;
}
RetryListener Class:
public class RetryListener implements IAnnotationTransformer {
#Override
public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
// TODO Auto-generated method stub
IRetryAnalyzer retry = annotation.getRetryAnalyzer();
if (retry == null) {
annotation.setRetryAnalyzer(TestRetry.class);
}
}}
SampleTest:
#Listeners(RetryListener.class)
public class SampleTest {
#BeforeSuite(alwaysRun = true)
public void beforeSuite(ITestContext context) {
for (ITestNGMethod method : context.getAllTestMethods()) {
method.setRetryAnalyzer(new TestRetry());
}
}
#Test(priority=0)
public void firsttest() {
System.out.println();
TestReporter.assertEquals("Test", "Test", "pass");
}
#Test(priority=1, dependsOnMethods="firsttest")
public void secondtest() {
TestReporter.assertEquals("Test", "Test1", "fail");
}
#Test(priority=2,dependsOnMethods="secondtest")
public void thirdtest() {
TestReporter.assertEquals("Test", "Test", "pass");
}
}
When I execute the above test, following is the output
firsttest gets executed and passes
secondtest depends on firsttest and gets executed, its failed - Retried 3 times and failed again
thirdtest skipped because it depends on secondtest.
Output achieved as expected.
Question:
Since the tests are dependent. If one of the tests fails, I want to execute the whole class from first. is there a way to do it?
Examples:
If secondtest fails, I want to execute the whole class SampleTest again.
Thanks!

There's currently no way of achieving what you are asking for.
TestNG will only retry a failed test, but will not go up the execution ladder to find out all the upstream dependencies and try running them as well (Your ask is a very specific variant of this generic use case).
If you come to think of it, a dependent test is being executed only because its upstream dependencies (methods on which it depends on) have been executed successfully. So if there's a failure in the current test, why would one need to re-execute the already satisfied upstream dependencies? Its counter intuitive.
For what you have as a use-case, you should be merely building the entire logic within a #Test method, wherein you take care of handling the retries and also the invocation of the entire chain once again, if there were failures.
The below sample should clarify that
public class SampleTest {
#Test (retryAnalyzer = TestRetry.class)
public void orchestrateTest() {
firsttest();
secondtest();
thirdtest();
}
public void firsttest() {
System.out.println();
TestReporter.assertEquals("Test", "Test", "pass");
}
public void secondtest() {
TestReporter.assertEquals("Test", "Test1", "fail");
}
public void thirdtest() {
TestReporter.assertEquals("Test", "Test", "pass");
}
}
TestNG does not support the use case that you are looking for in your question.
On a side note, you cannot wire in a IAnnotationTransformer listener via an #Listeners annotation (this is explicitly called out in the javadocs of this interface). It can only be wired in via the <listeners> tag in your suite xml (or) by referring to it in the META-INF\services\org.testng.ITestNGListener file (its called the Service Provider Interface approach in Java)

Related

JUnit5: Before and After Suite method invocation

I've spent a few days trying to find out a solution, so I know all the basic answers
I've read the documentation, and I know, that #BeforeClass/#AfterClass is replaced with #BeforeAll/#AfterAll, #RunWith no longer exists; superseded by #ExtendWith
I've read all the topics here on the stackoverflow (removed links, since stackoverflow pretends it's a spam:(
I used to run some JUnit4 tests in the suites. Each Suite class Suite1.class, Suite2.class had several #Test methods and #BeforeClass/#AfterClass were running exactly before/after all the testing methods.
#RunWith(StopOnFailureSuite.class)
#Suite.SuiteClasses({
Test1.class,
Test2.class
})
public class TSuite_20 {
private static final byte SUITE_NUMBER = 20;
#BeforeClass
public static void setUpClass() {
//some logic for suite setup
}
#AfterClass
public static void tearDownClass() {
//some logic for teardown
}
}
As I wanted to use #ParameterizedTests I need to migrate to JUnit5.
And suddenly I realised, that exact the same behaviour, that used to be in JUnit4 is no more achievable.
[run some custom setup code; run several test classes, which may contain several #Test methods; run some custom tear down code];
Does anybody know (better with examples) an approach to make it with JUnit 5?
Option 1
This code will never execute BeforeAfterSuite#beforeAll and BeforeAfterSuite#afterAll
#ExtendWith(BeforeAfterSuite.class)
#Suite
#SelectClasses({
Test1.class,
Test2.class
})
public class TSuite_20 {
public static final byte SUITE_NUMBER = 20;
}
public class BeforeAfterSuite implements BeforeAllCallback, AfterAllCallback
/*,TestInstancePreConstructCallback, BeforeTestExecutionCallback,
AfterTestExecutionCallback, ExtensionContext.Store.CloseableResource*/ {
private static boolean started = false;
#Override
public void beforeAll(ExtensionContext context) {
if (!started) {
started = true;
//before stuff
}
}
#Override
public void afterAll(ExtensionContext context) throws Exception {
//after all;
}
}
Option 2
I was just curious, how will JUnit treat suite class if I put a test method into it...
This code will execute BeforeAfterSuite#beforeAll and BeforeAfterSuite#afterAll once, before and after TSuite_20#test
#ExtendWith(BeforeAfterSuite.class)
#Suite
#SelectClasses({
Test1.class,
Test2.class
})
public class TSuite_20 {
public static final byte SUITE_NUMBER = 20;
#Test
public void test() {
}
}
Option 3
We also can apply #ExtendWith(BeforeAfterSuite.class) per Test class which will results in a BeforeAfterSuite#beforeAll and BeforeAfterSuite#afterAll per Test class. (in this example - 2 times).
#ExtendWith(BeforeAfterSuite.class)
public class Test1 {
#Test
public void test11() {
}
#Test
public void test12() {
}
}
#ExtendWith(BeforeAfterSuite.class)
public class Test2 {
#Test
public void test21() {
}
#Test
public void test22() {
}
}
Option 4
I also give a shot for
a Suite class without #ExtendWith() and #BeforeAll + #AfterAll; (as expected nothing happened)`
a Suite class without #ExtendWith() and #BeforeAll + #Test + #AfterAll; (as expected single execution of BeforeAll/AfterAll for the specific Suite class)`
Option 5
Listeners were my last hope to achieve the desired behaviour.
I've created my own impl for LauncherSessionListener, just because I've thought it will allow me to execute smth exactly before tests start.
public class BeforeAfterSuiteLauncher implements LauncherSessionListener {
private static boolean started = false;
#Override
public void launcherSessionOpened(LauncherSession session) {
if (!started) {
started = true;
//before all
}
}
#Override
public void launcherSessionClosed(LauncherSession session) {
//after all
}
}
And I've added also some default impl CompositeLauncherSessionListener
Packages structure screenshot to show Java SPI configuration: LauncherSessionListener
For TestExecutionListener I've added two default impls, just to catch at least one Listener:
org.junit.platform.launcher.listeners.LoggingListener
org.junit.platform.launcher.listeners.SummaryGeneratingListener
and one custom
public class BeforeAfterExecutionListener implements TestExecutionListener {
#Override
public void testPlanExecutionStarted(TestPlan testPlan) {
//before all
}
#Override
public void testPlanExecutionFinished(TestPlan testPlan) {
//after all
}
}
Packages structure screenshot to show Java SPI configuration:TestExecutionListener
And only SummaryGeneratingListener was triggered!
What am I doing wrong? Why my BeforeAfterExecutionListener impl was not loaded and triggered?
P.S. All of the above code was executed under Intellij Idea 2021.1.3 Ultimate Edition
java version "1.8.0_341"
Java(TM) SE Runtime Environment (build 1.8.0_341-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.341-b10, mixed mode)
here is intelliJs command:
C:\Tools\jdk\bin\java.exe -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:61280,suspend=y,server=n -ea -Didea.test.cyclic.buffer.size=1048576 -javaagent:C:\Users\userName\AppData\Local\JetBrains\IntelliJIdea2021.1\groovyHotSwap\gragent.jar -javaagent:C:\Users\userName\AppData\Local\JetBrains\IntelliJIdea2021.1\captureAgent\debugger-agent.jar -Dfile.encoding=UTF-8 -classpath C:\Users\userName\AppData\Local\Temp\classpath1705687115.jar com.intellij.rt.junit.JUnitStarter -ideVersion5 -junit5 com.testdirectly.application.testcase.TSuite_20
Which results in JUnit5IdeaTestRunner
My gradle dependencies
dependencies {
//JUnit platform
// to group tests by package, by class name, by class name pattern, etc (use #Suite, #SelectClasses) :junit-platform-suite-api:1.9.2
// and to filter/discover and run them (SuiteLauncher, SuiteTestEngine, SuiteTestDescriptor) :junit-suite-engine:1.9.2
testImplementation "org.junit.platform:junit-platform-suite:1.9.2"
//Launcher, engine discovery
testImplementation "org.junit.platform:junit-platform-launcher:1.9.2"//to run tests
//JUnit Jupiter
//to use assertions and so on
testImplementation "org.junit.jupiter:junit-jupiter-api:5.9.2"
//to use #ParameterizedTest
testImplementation "org.junit.jupiter:junit-jupiter-params:5.9.2"
//Jupiter engine to run junit5 tests (JupiterTestEngine, Extensions, etc)
testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.9.2"
}
Short answer
Option 5 works if you tweak it slightly (see below).
Long answer
Options 1 and 2 do not work because #ExtendWith is a Jupiter extension mechanism whereas #Suite triggers an engine of its own.
Test engines do not combine on the JUnit platform.
The same goes for option 4 since #BeforeAll and #AfterAll are Jupiter annotations.
Making Option 5 Work
First of all I'd suggest to use TestExecutionListener instead of LauncherSessionListener because the latter is still experimental.
Thus we have
package my.project.suites;
import org.junit.platform.launcher.*;
public class BeforeAfterSuiteListener implements TestExecutionListener {
#Override
public void testPlanExecutionStarted(TestPlan testPlan) {
System.out.println("before all");
}
#Override
public void testPlanExecutionFinished(TestPlan testPlan) {
System.out.println("after all");
}
}
The missing thing is now that you'll have to register BeforeAfterSuiteListener globally.
In classpath-based Java you do that through a resource file
META-INF/services/org.junit.platform.launcher.TestExecutionListener:
my.project.suites.BeforeAfterSuiteListener
Now before all and after all should show up in your output exactly once per test run.

TestNG, is there a class level listener like IClassListener like ITestListener

I want to perform same action for every class (just like #BeforeClass). I guess listeners can do things where you don't have to write code individually, but I did not find in each method/class but can be executed via a listener. Is there a way to execute my method before every class or just once before method of that class?
Check the beforeConfiguration() method in TestListenerAdapter.
#Override
public void beforeConfiguration(ITestResult tr) {
if(tr.getMethod().getMethodName().equals("methodNameForBeforeClass")) {
//...
}
}
Try configuration related methods in TestListenerAdapter:
class TestNGListener extends TestListenerAdapter {
#Override
public void beforeConfiguration(ITestResult tr) {
super.beforeConfiguration(tr);
logger.info("=========== Configuration method '{}' started ===========", tr.getMethod().getMethodName());
}
#Override
public void onConfigurationSuccess(ITestResult tr) {
super.onConfigurationSuccess(tr);
logger.info("=========== Configuration method '{}' finished ===========", tr.getMethod().getMethodName());
}
#Override
public void onConfigurationFailure(ITestResult tr) {
super.onConfigurationFailure(tr);
logger.error("!!!!!!!!!!! Configuration method '{}' failed !!!!!!!!!!!", tr.getMethod().getMethodName());
}
}
Extend TestListenerAdapter and override onTestStart(ITestResult result) method. This will help you to run something everytime a test starts

How to test an NServiceBus handler that uses Schedule

I'm playing with a simple handler that implements IWantToRunWhenBusStartsAndStops and in the start, it schedules a task like so:
public void Start()
{
_schedule.Every(TimeSpan.FromSeconds(5), Moo);
}
_schedule is injected via the constructor. The test I'm trying write is to make sure the task is scheduled when the handler starts. But I can't find a way to mock Schedule as it doesn't have a no-arg constructor and it doesn't implement an interface. I tried creating an actual instance of it with a mocked IBuilder but can't figure out what expectations to set on the IBuilder. Also, I looked at the source to see how they were testing Schedule but it looks like we're on an earlier version (v5.0.0 via nuget) because we don't have a DefaultScheduler which appears to be what they use in their current tests.
In fact NServiceBus team has already covered the scheduler with unit/acceptance test, i.e. there is no need to check whether the task was actually scheduled when your handler is executed. Instead you would probably want to unit test your handler itself, thus check if call to scheduler.Every() has been made. Here is simple example of how your unit test might look like:
[TestClass]
public class Tests
{
[TestMethod]
public void When_executing_handler_the_task_should_be_scheduled()
{
//arrange
var scheduler = new FakeSheduler();
//act
var handler = new TestHandler(scheduler);
handler.Start();
//assert
Assert.IsTrue(scheduler.WasCalled);
}
}
The handler itself:
class TestHandler: IWantToRunWhenBusStartsAndStops
{
readonly IMyScheduler _scheduler;
public TestHandler(IMyScheduler scheduler)
{
_scheduler = scheduler;
}
public void Start()
{
_scheduler.Every(TimeSpan.FromSeconds(5), () => { });
}
public void Stop() { }
}
Finally, you have to abstract from direct usage of NServiceBus scheduler in order to make it testable, here is the idea:
interface IMyScheduler
{
void Every(TimeSpan interval, Action action);
}
//your real implementation
class MySheduler: IMyScheduler
{
readonly Schedule _schedule;
public MySheduler(Schedule schedule)
{
_schedule = schedule;
}
public void Every(TimeSpan interval, Action action)
{
_schedule.Every(TimeSpan.FromSeconds(5), () => { });
}
}
//fake for the testing
class FakeSheduler: IMyScheduler
{
public bool WasCalled { get; set; }
public void Every(TimeSpan interval, Action action)
{
WasCalled = true;
}
}

Jmockit Expectations/Verifications for calls to private methods when testing a public method?

Can anyone advise if it is possible to use an expectations/verifications to test that private methods are being called the-right-number-of-times/right-parameters.
The Class under test has been Mocked-Up - with one private method overridden.
Am Testing a public method which calls into a number of private methods.
I wish to know if it is possible to verify the calls to other private methods which will be called when the public method is being executed ?
Some idea of the code/class under test;
public class UnderTest {
public void methodPublic(arg 1){
.....
methodPrivate1(var1);
....
methodPrivate2(var2);
}
private void methodPrivate1(var1){
//do stuff
}
private void methodPrivate2(var1){
//do stuff
}
}
In my test case
#Test
public void stateBasedTestMethod()
{
UnderTest underTest;
new MockUp<UnderTest>() {
#Mock(invocations = 1)
private void methodPrivate2(var1) {
//do nothing in the mocked case
}
};
underTest = new UnderTest();
underTest.methodPublic(arg1);
new Verifications() {{
// Is there a way to test that methodPrivate1 has been called-once/with-expected-arguments
}};
}
Edited in response to the answer from Rogério.
I am using jmockit 1.12
and the Verifications is FAILING as the method using the provided solution is invoking the method twice as I thought from the JMockit documentation.
Failure Trace;
mockit.internal.UnexpectedInvocation: Expected exactly 1 invocation(s) of MyHelperTest$1#method3..., but was invoked 2 time(s)
Included is the full code I am using for this.
As described above - my goal is to mock one of the private methods to do nothing.
And ensure that I can verify that the other private method is called only once.
Thanks in advance and hopefully will get a better understanding if this is possible with Jmockit.
Test Code.
public class MyHelperTest {
#Test
public void testHelper(#Mocked final MyDependent myDependent) {
final MyHelper myHelper;
new MockUp<MyHelper>() {
#Mock(invocations = 1)
private void method3(MyDependent myTable) {
System.out.println("In Mocked Method");
//do nothing in the mocked case
}
};
myHelper = new MyHelper();
myHelper.method1(myDependent);
new Verifications() {{
invoke(myHelper, "method2", myDependent); times = 1;
}};
}
}
Class under test.
public class MyHelper {
public void method1(MyDependent myDependent){
method2(myDependent);
}
private void method2(MyDependent myDependent) {
myDependent.setValue(1);
method3(myDependent);
}
private void method3(MyDependent myDependent) {
myDependent.setValue(2);
}
}
Dependent Class
public class MyDependent {
private int value;
public int getValue() {
return value;
}
public void setValue(int value) {
this.value = value;
}
}
It's possible, though not recommended to mock private methods.
Using the Expectations API:
#Tested #Mocked MyHelper myHelper;
#Test
public void testHelper(#Mocked final MyDependent myDependent)
{
new NonStrictExpectations() {{ invoke(myHelper, "method3", myDependent); }};
myHelper.method1(myDependent);
new Verifications() {{ invoke(myHelper, "method2", myDependent); times = 1; }};
}
... where the invoke(...) method is statically imported from class mockit.Deencapsulation.
I noticed that if a method you want to verify is not mocked, when the static block in an Expectations or Verifications instance is executed that the code calls the method that you are trying to mark as expected or verify.
This might explain the extra invocation that you are seeing.
One suggestion: if you are already mocking the class with MockUp (and thus creating an anonymous subclass) so you can override the private method, why not change the access of the overridden private method to protected or public? Then you can create an expectation or verification on it.
You could also provide a public field "public int counter=0;" and have your overridden method increment the counter. Then you can use an assert on it after the test is complete.

Google Guice, Interceptors and PrivateModules

New poster here, hope I don't brake any rules :)
I am using PrivateModule in google-guice in order to have multiple DataSource's for the same environment. But I am having a hard time getting MethodInterceptor's to work inside the private modules.
Below is a simple test case that explains the "problem".
A simple service class would be:
interface Service {
String go();
}
class ServiceImpl implements Service {
#Override #Transactional
public String go() {
return "Test Case...";
}
}
The MyModule class would be:
class MyModule extends AbstractModule {
#Override
protected void configure() {
install(new PrivateModule() {
#Override
protected void configure() {
bind(Service.class).to(ServiceImpl.class);
bindInterceptor(
Matchers.any(),
Matchers.annotatedWith(Transactional.class),
new MethodInterceptor() {
#Override
public Object invoke(MethodInvocation i)
throws Throwable {
System.out.println("Intercepting: "
+ i.getMethod().getName());
return i.proceed();
}
});
expose(Service.class);
}
});
}
}
And the final test case:
public class TestCase {
#Inject Service service;
public TestCase() {
Guice.createInjector(new MyModule()).injectMembers(this);
}
public String go() {
return service.go();
}
public static void main(String[] args) {
TestCase t = new TestCase();
System.out.println(t.go());
}
}
You would expect the output to be:
Intercepting: go
Test Case...
But it doesn't happen, the interceptor is not used, ant only Test Case... is output.
If I bind/expose the ServiceImpl instead of the interface then it works.
Thanks in advance,
Regards,
LL
Well... I figured it out shortly after I posted the question :)
The problem is that you also need to expose() the ServiceImpl class.
So the bind/expose would be.
bind(ServiceImpl.class); // ServiceImpl annotated with #Singleton
bind(Service.class).to(ServiceImpl.class);
expose(ServiceImpl.class);
expose(Service.class);
Regards,
LL
You need to explicitly bind ServiceImpl in the private module. The problem with your existing code is that it inherits the binding for ServiceImpl from the parent module. From the PrivateModule docs,
Private modules are implemented using parent injectors. When it can satisfy their dependencies, just-in-time bindings will be created in the root environment. Such bindings are shared among all environments in the tree.
Adding this line should fix the problem:
bind(ServiceImpl.class);