I'm having some problems with the Priority annotation in TestNG not running test cases. My priority numbers are not sequential, I have intentionally skipped numbers so at a future date I can add some test methods between them so the run in a specific order. Any ideas how I can get TestNG to execute my test methods that follow the number gap? For example, I have priorities in the order of 1 - 3 and then I jump to 200 - 202. The tests with priorities 200 - 202 are not being executed. Thanks for your time.
public class ClassOneTest{
#Test (enabled=true, priority=1)
public void methodOneTest() {
.....
}
/**
* Updates the ServiceNow End Point
*/
#Test (enabled=true, priority=2)
public void methodTwoTest() {
.....
}
/**
* Deletes the ServiceNow End Point
*/
#Test (enabled=true, priority=3)
public void methodTwoTest() {
.....
}
}
public class ClassTwoTest{
#Test (enabled=true, priority=200)
public void methodThreeTest() {
.....
}
/**
* Updates the ServiceNow End Point
*/
#Test (enabled=true, priority=201)
public void methodFourTest() {
.....
}
/**
* Deletes the ServiceNow End Point
*/
#Test (enabled=true, priority=202)
public void methodFiveTest() {
.....
}
}
I found the problem. The class with the priorities in the 200's was being loaded in the suite before the class with the lower priorities. Once I changed the loading order it worked fine.
Related
I've spent a few days trying to find out a solution, so I know all the basic answers
I've read the documentation, and I know, that #BeforeClass/#AfterClass is replaced with #BeforeAll/#AfterAll, #RunWith no longer exists; superseded by #ExtendWith
I've read all the topics here on the stackoverflow (removed links, since stackoverflow pretends it's a spam:(
I used to run some JUnit4 tests in the suites. Each Suite class Suite1.class, Suite2.class had several #Test methods and #BeforeClass/#AfterClass were running exactly before/after all the testing methods.
#RunWith(StopOnFailureSuite.class)
#Suite.SuiteClasses({
Test1.class,
Test2.class
})
public class TSuite_20 {
private static final byte SUITE_NUMBER = 20;
#BeforeClass
public static void setUpClass() {
//some logic for suite setup
}
#AfterClass
public static void tearDownClass() {
//some logic for teardown
}
}
As I wanted to use #ParameterizedTests I need to migrate to JUnit5.
And suddenly I realised, that exact the same behaviour, that used to be in JUnit4 is no more achievable.
[run some custom setup code; run several test classes, which may contain several #Test methods; run some custom tear down code];
Does anybody know (better with examples) an approach to make it with JUnit 5?
Option 1
This code will never execute BeforeAfterSuite#beforeAll and BeforeAfterSuite#afterAll
#ExtendWith(BeforeAfterSuite.class)
#Suite
#SelectClasses({
Test1.class,
Test2.class
})
public class TSuite_20 {
public static final byte SUITE_NUMBER = 20;
}
public class BeforeAfterSuite implements BeforeAllCallback, AfterAllCallback
/*,TestInstancePreConstructCallback, BeforeTestExecutionCallback,
AfterTestExecutionCallback, ExtensionContext.Store.CloseableResource*/ {
private static boolean started = false;
#Override
public void beforeAll(ExtensionContext context) {
if (!started) {
started = true;
//before stuff
}
}
#Override
public void afterAll(ExtensionContext context) throws Exception {
//after all;
}
}
Option 2
I was just curious, how will JUnit treat suite class if I put a test method into it...
This code will execute BeforeAfterSuite#beforeAll and BeforeAfterSuite#afterAll once, before and after TSuite_20#test
#ExtendWith(BeforeAfterSuite.class)
#Suite
#SelectClasses({
Test1.class,
Test2.class
})
public class TSuite_20 {
public static final byte SUITE_NUMBER = 20;
#Test
public void test() {
}
}
Option 3
We also can apply #ExtendWith(BeforeAfterSuite.class) per Test class which will results in a BeforeAfterSuite#beforeAll and BeforeAfterSuite#afterAll per Test class. (in this example - 2 times).
#ExtendWith(BeforeAfterSuite.class)
public class Test1 {
#Test
public void test11() {
}
#Test
public void test12() {
}
}
#ExtendWith(BeforeAfterSuite.class)
public class Test2 {
#Test
public void test21() {
}
#Test
public void test22() {
}
}
Option 4
I also give a shot for
a Suite class without #ExtendWith() and #BeforeAll + #AfterAll; (as expected nothing happened)`
a Suite class without #ExtendWith() and #BeforeAll + #Test + #AfterAll; (as expected single execution of BeforeAll/AfterAll for the specific Suite class)`
Option 5
Listeners were my last hope to achieve the desired behaviour.
I've created my own impl for LauncherSessionListener, just because I've thought it will allow me to execute smth exactly before tests start.
public class BeforeAfterSuiteLauncher implements LauncherSessionListener {
private static boolean started = false;
#Override
public void launcherSessionOpened(LauncherSession session) {
if (!started) {
started = true;
//before all
}
}
#Override
public void launcherSessionClosed(LauncherSession session) {
//after all
}
}
And I've added also some default impl CompositeLauncherSessionListener
Packages structure screenshot to show Java SPI configuration: LauncherSessionListener
For TestExecutionListener I've added two default impls, just to catch at least one Listener:
org.junit.platform.launcher.listeners.LoggingListener
org.junit.platform.launcher.listeners.SummaryGeneratingListener
and one custom
public class BeforeAfterExecutionListener implements TestExecutionListener {
#Override
public void testPlanExecutionStarted(TestPlan testPlan) {
//before all
}
#Override
public void testPlanExecutionFinished(TestPlan testPlan) {
//after all
}
}
Packages structure screenshot to show Java SPI configuration:TestExecutionListener
And only SummaryGeneratingListener was triggered!
What am I doing wrong? Why my BeforeAfterExecutionListener impl was not loaded and triggered?
P.S. All of the above code was executed under Intellij Idea 2021.1.3 Ultimate Edition
java version "1.8.0_341"
Java(TM) SE Runtime Environment (build 1.8.0_341-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.341-b10, mixed mode)
here is intelliJs command:
C:\Tools\jdk\bin\java.exe -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:61280,suspend=y,server=n -ea -Didea.test.cyclic.buffer.size=1048576 -javaagent:C:\Users\userName\AppData\Local\JetBrains\IntelliJIdea2021.1\groovyHotSwap\gragent.jar -javaagent:C:\Users\userName\AppData\Local\JetBrains\IntelliJIdea2021.1\captureAgent\debugger-agent.jar -Dfile.encoding=UTF-8 -classpath C:\Users\userName\AppData\Local\Temp\classpath1705687115.jar com.intellij.rt.junit.JUnitStarter -ideVersion5 -junit5 com.testdirectly.application.testcase.TSuite_20
Which results in JUnit5IdeaTestRunner
My gradle dependencies
dependencies {
//JUnit platform
// to group tests by package, by class name, by class name pattern, etc (use #Suite, #SelectClasses) :junit-platform-suite-api:1.9.2
// and to filter/discover and run them (SuiteLauncher, SuiteTestEngine, SuiteTestDescriptor) :junit-suite-engine:1.9.2
testImplementation "org.junit.platform:junit-platform-suite:1.9.2"
//Launcher, engine discovery
testImplementation "org.junit.platform:junit-platform-launcher:1.9.2"//to run tests
//JUnit Jupiter
//to use assertions and so on
testImplementation "org.junit.jupiter:junit-jupiter-api:5.9.2"
//to use #ParameterizedTest
testImplementation "org.junit.jupiter:junit-jupiter-params:5.9.2"
//Jupiter engine to run junit5 tests (JupiterTestEngine, Extensions, etc)
testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.9.2"
}
Short answer
Option 5 works if you tweak it slightly (see below).
Long answer
Options 1 and 2 do not work because #ExtendWith is a Jupiter extension mechanism whereas #Suite triggers an engine of its own.
Test engines do not combine on the JUnit platform.
The same goes for option 4 since #BeforeAll and #AfterAll are Jupiter annotations.
Making Option 5 Work
First of all I'd suggest to use TestExecutionListener instead of LauncherSessionListener because the latter is still experimental.
Thus we have
package my.project.suites;
import org.junit.platform.launcher.*;
public class BeforeAfterSuiteListener implements TestExecutionListener {
#Override
public void testPlanExecutionStarted(TestPlan testPlan) {
System.out.println("before all");
}
#Override
public void testPlanExecutionFinished(TestPlan testPlan) {
System.out.println("after all");
}
}
The missing thing is now that you'll have to register BeforeAfterSuiteListener globally.
In classpath-based Java you do that through a resource file
META-INF/services/org.junit.platform.launcher.TestExecutionListener:
my.project.suites.BeforeAfterSuiteListener
Now before all and after all should show up in your output exactly once per test run.
Is it possible to run migration and seeding once and don't refresh the testing db between the test methods?
I have couple of testing functions that depend on each other and I don't want to migrate and seed the database before and after each test in one testing file.
Example:
<?php
namespace Tests\Browser;
use Tests\DuskTestCase;
use Laravel\Dusk\Browser;
use Illuminate\Foundation\Testing\DatabaseMigrations;
use Carbon\Carbon;
class AdminTest extends DuskTestCase
{
use DatabaseMigrations;
/**
* Define hooks to migrate the database before and after each test.
*
* #return void
*/
protected function setUp(): void
{
parent::setUp();
$this->artisan('db:seed', ['--class' => 'DatabaseSeeder']);
}
public function testAdminCanLogin()
{
}
/* Create New ticket */
public function testAdminCreateTicket()
{
}
/* View the first ticket */
public function testAdminViewTicket()
{
}
/* Edit the first ticket */
public function testAdminEditTicket()
{
}
/* Assign the first Ticket to an Agent */
public function testAdminAssignTicketToAgent()
{
}
/* Unassign the first Ticket from Agent */
public function testAdminUnassignAgentFromTicket()
{
}
/* Delete the first ticket */
public function testAdminDeleteTicket()
{
}
/* Restore the first ticket */
public function testAdminRestoreTicket()
{
}
}
Yes, You can do something like this
protected static $migrationRun = false;
public function setUp(): void{
parent::setUp();
if(!static::$migrationRun){
$this->artisan('migrate:refresh');
$this->artisan('db:seed');
static::$migrationRun = true;
}
}
Include this in your dusk test class. setUp method runs before each test method, If migration has run once, It won't run it again.
don't use use DatabaseMigrations.
just:
$this->artisan('migrate:fresh');
$this->artisan('db:seed');
like:
public function setUp(): void
{
$this->appUrl = env('APP_URL');
parent::setUp();
$this->artisan('migrate:fresh');
$this->artisan('db:seed');
}
in your first browser test
I am using testNG for my selenium suite. There is a class having 35 test cases. But these test cases will execute only if a particular element is visible. If that element is not visible, the compiler goes through all the test cases. Is there any way that I could check that element visibility condition in an #BeforeClass annotation only. If an element is not visible, it should come out from that class and switch to the next one? It will save my time to go through all the test cases?
To achieve it use #Test annotation on class level and #BeforeTest to check element visibility so it will skip all test cases of class if it will not satisfy condition in #BeforeTest. See below code (it's tested and working).
#Test
public class SkipAllTestCases {
boolean elementNotVisible=true;
#BeforeTest
public void setUp() {
if (elementNotVisible) {
throw new SkipException("skipping test cases...");
}
}
public void test1() {
System.out.println("Test1");
}
public void test2() {
System.out.println("Test2");
}
public void test3() {
System.out.println("Test3");
}
}
Hope it will help.
You can use dependsOnMethods of TestNG Test annotation.
#Test
public void elementVisibleTest(){
//Fail or skip here
}
#Test(dependsOnMethods = {"elementVisibleTest"})
public void myOtherTest(){
//Do something
}
...
That means if elementVisibleTest fails or gets skipped all tests which depend on that test will be skipped too. The advantage of that would be that you can still have other tests in that class which will be executed (because they do not depend on elementVisibleTest).
One of the approach is add group to all such tests let say flow-1. Add before group method and throw exception if it doesn't match required condition. For example:
#BeforeGroups(groups="flow-1")
public void flow1() {
if(!requiredCondtionMatch) {
throw new SkipException("Flow not applicable");
}
}
If all tests falls under same class then you can use #BeforeClass as well.
Following are the classes used to implement retry logic
TestRetry Class:
public class TestRetry implements IRetryAnalyzer {
int counter=0;
int retryLimit=2;
#Override
public boolean retry(ITestResult result) {
if (counter < retryLimit) {
TestReporter.logStep("Retrying Test " +result.getName()+" for number of times: "+(counter+1));
counter++;
return true;
}
return false;
}
RetryListener Class:
public class RetryListener implements IAnnotationTransformer {
#Override
public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
// TODO Auto-generated method stub
IRetryAnalyzer retry = annotation.getRetryAnalyzer();
if (retry == null) {
annotation.setRetryAnalyzer(TestRetry.class);
}
}}
SampleTest:
#Listeners(RetryListener.class)
public class SampleTest {
#BeforeSuite(alwaysRun = true)
public void beforeSuite(ITestContext context) {
for (ITestNGMethod method : context.getAllTestMethods()) {
method.setRetryAnalyzer(new TestRetry());
}
}
#Test(priority=0)
public void firsttest() {
System.out.println();
TestReporter.assertEquals("Test", "Test", "pass");
}
#Test(priority=1, dependsOnMethods="firsttest")
public void secondtest() {
TestReporter.assertEquals("Test", "Test1", "fail");
}
#Test(priority=2,dependsOnMethods="secondtest")
public void thirdtest() {
TestReporter.assertEquals("Test", "Test", "pass");
}
}
When I execute the above test, following is the output
firsttest gets executed and passes
secondtest depends on firsttest and gets executed, its failed - Retried 3 times and failed again
thirdtest skipped because it depends on secondtest.
Output achieved as expected.
Question:
Since the tests are dependent. If one of the tests fails, I want to execute the whole class from first. is there a way to do it?
Examples:
If secondtest fails, I want to execute the whole class SampleTest again.
Thanks!
There's currently no way of achieving what you are asking for.
TestNG will only retry a failed test, but will not go up the execution ladder to find out all the upstream dependencies and try running them as well (Your ask is a very specific variant of this generic use case).
If you come to think of it, a dependent test is being executed only because its upstream dependencies (methods on which it depends on) have been executed successfully. So if there's a failure in the current test, why would one need to re-execute the already satisfied upstream dependencies? Its counter intuitive.
For what you have as a use-case, you should be merely building the entire logic within a #Test method, wherein you take care of handling the retries and also the invocation of the entire chain once again, if there were failures.
The below sample should clarify that
public class SampleTest {
#Test (retryAnalyzer = TestRetry.class)
public void orchestrateTest() {
firsttest();
secondtest();
thirdtest();
}
public void firsttest() {
System.out.println();
TestReporter.assertEquals("Test", "Test", "pass");
}
public void secondtest() {
TestReporter.assertEquals("Test", "Test1", "fail");
}
public void thirdtest() {
TestReporter.assertEquals("Test", "Test", "pass");
}
}
TestNG does not support the use case that you are looking for in your question.
On a side note, you cannot wire in a IAnnotationTransformer listener via an #Listeners annotation (this is explicitly called out in the javadocs of this interface). It can only be wired in via the <listeners> tag in your suite xml (or) by referring to it in the META-INF\services\org.testng.ITestNGListener file (its called the Service Provider Interface approach in Java)
Here's the code. The code in method test and test2 are different because the parameter passed to Test constructor are different. Actually, if I change any parameter to null, intellij stops reporting the duplication. Is there any way to fix this?
---- Updated --------
I pass 2 functions doing totally different things but intellij still reports duplication
public class TestMain {
public void test(int a)
{
System.out.println("haha");
System.out.println("hahaa");
TestMain testMain = new TestMain();
new Test(testMain::test3);
System.out.println("hahaaa");
}
public void test2(int a)
{
System.out.println("haha");
System.out.println("hahaa");
TestMain testMain = new TestMain();
new Test(testMain::still_dup);
System.out.println("hahaaa");
}
public void test3(int a) {
System.out.println("abc");
}
public void still_dup(int a) {
String b = "edf";
b.toLowerCase();
}
public class Test {
Test(handler h) {
}
}
public interface handler<M> {
void entitySelector(int a);
}
public static void main(String[] args) {
TestMain test = new TestMain();
test.test(1);
System.out.println("-------");
test.test2(2);
}
}
I think the best way to fix this is to replace test and test2 by a single method. You don't have to distinguish what to pass the constructor because it's the current method. This might be the reason why code duplication is reported. The methods can be replaced by a single one without problems.